Distributed Algorithms for Constructing Balanced Spanning Trees on System-ranked Process Groups

Parallel programs often express operations on a subset (process group) of all the participating processes or ranks. Subcommunicators in MPI are an example of such process groups. Often, these process groups are used only for simple collective communication (broadcast, reduction, allreduce) over the members of the process group. Current algorithms to create process groups tend to be centralized schemes which store or manipulate data structures of size proportional to the number of members in the process group. In extreme scale parallel architectures, these algorithms will consume a prohibitive amount of memory to manage the process group and also suffer from scalability limits. In this paper, we contend that MPI subcommunicators pack more features than are necessary for a sizeable fraction of the use cases. We make a case for system-ranked process groups, intended primarily for simple collective operations. As opposed to MPI communicators, member ranks in system-ranked process groups are assigned by the runtime system. This paper presents distributed algorithms for the creation of spanning trees for rudimentary collective communication over system-ranked process groups that are obtained by splitting an original parent process group. Our schemes use only a small constant amount of memory per node and also perform better than a reference centralized scheme even at modest process counts. We also demonstrate performance results up to 131, 072 cores of BlueGene/P. Our algorithms can apply to the creation of MPI subcommunicators as well as to equivalent entities in other parallel programming paradigms. Keywords-sub-communicators, spanning trees, distributed algorithms, exascale